Challenges in Ranking of Universities

In this paper we discuss recent developments in rankings of universities and the impact of these rankings on academia in the context of international benchmarking and evaluation. We focus on technical and methodological problems behind these rankings, particularly those based on bibliometric methods, with special attention to the social sciences and humanities. We criticize the recent expert-based rankings by showing that the correlation of expert score and bibliometric outcomes is practically zero. This finding casts severe doubts on the reliability of these expertbased rankings. New approaches are proposed on the basis of advanced bibliometric methods. It is argued on the basis of preliminary results that, probably due to ‘finite size’ considerations, a league of outstanding universities worldwide will not have much more than around 200 members. Finally, we discuss the challenges for further research and practical applications. 1. Context and Central Questions Scientific research has always been an international enterprise. The growth in the last few decades of worldwide R&D activities as well as the higher education sector in developed and developing countries reinforced established academic institutions and created many new ones. At the same time, the number and intensity of student and researcher exchange programs, international collaboration, and working stays outside the own country rapidly increased, intensified by the evergrowing worldwide mobility. An important parallel development, stimulated too by the growth of the higher education sector, is the strong demand for accountability, evidence of quality, and ‘value for money’. This latter development gave rise to an expanding evaluation culture. This evaluation culture is based on elements present for already a very long time in the scientific world such as judgment of research quality by peer review. All these developments led to an increasing competition for financial support and for the best students and researchers between universities within nations and worldwide. Universities strive for belonging to the top of their country, or even to the world top. This implies the existence of some kind of a league (a national one, or an international one), to which one can only be admitted on the basis of performance. The higher the performance, the better chances a university has to become a member of an elite league and to reach a high ranking position in this league. Clearly, the basic question is: How can we identify the best universities in the world? From this basic question, we derive several research questions that have to be answered in the context of ranking procedures. First, to which activities are the leagues related? In the case of